Determination of position of a head-mounted device on a user

ABSTRACT

There is provided a method and system for determining if a head-mounted device for extended reality (XR) is correctly positioned on a user, and optionally performing a position correction procedure if the head-mounted device is determined to be incorrectly positioned on the user. Embodiments include: performing eye tracking by estimating, based on a first image of a first eye of the user, a position of a pupil in two dimensions; determining whether the estimated position of the pupil of the first eye is within a predetermined allowable area in the first image; and, if the determined position of the pupil of the first eye is inside the predetermined allowable area, concluding that the head-mounted device is correctly positioned on the user; or, if the determined position of the pupil of the first eye is outside the predetermined allowable area, concluding that the head-mounted device is incorrectly positioned on the user.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application claims priority to Swedish Application No. 1851340-8,filed Oct. 29, 2018; the content of which are hereby incorporated byreference.

TECHNICAL FIELD

The present disclosure generally relates to determining the position ofa head-mounted device for virtual reality (VR), augmented reality (AR),mixed reality (MR) or other extended reality (XR) on a user. The presentdisclosure also relates to corresponding systems and storage media.

BACKGROUND

The performance of head-mounted devices such as virtual-reality (VR)headsets, augmented reality (AR) headsets, mixed reality (MR) headsetsor other version of extended reality (XR) headset may be negativelyaffected if the head-mounted device is not properly mounted at theuser's head. Also, even if a head-mounted device has initially beenpositioned correctly, the head-mounted device may be repositioned duringuse, for example due to slippage when the head of the user moves orrotates.

Since lenses or other optical equipment of the head-mounted device aretypically optimized for a certain position relative to the user's eyes,a user wearing the head-mounted device incorrectly may experienceproblems that lead to an inferior visual experience. For example, if thehead-mounted device is adapted to provide a three-dimensional (3D)experience to the user, this 3D experience may become distorted if thehead-mounted device is incorrectly positioned on the user's head. A VRor MR experience provided by the head-mounted device may for examplebecome less realistic, or virtual objects shown by an AR or MR headsetmay not fit into the real environment as well as they could have done.Any images shown to the user on a screen of the head-mounted device mayfurther be experienced as blurry.

If the head-mounted device is equipped with eye tracking or gazetracking functionality, the performance of such functionality may alsobe negatively affected if the head-mounted device is incorrectlypositioned on the user's head, since the eye tracker or gaze trackerwill not be able to see/detect the user's eyes, and hence the quality ofany tracking signals will be poor.

There is a need to provide solutions to one or more of the identifiedproblems.

SUMMARY

Methods, systems and computer-readable storage media having the featuresdefined in the independent claims are provided for solving or at leastameliorating one or more of the identified problems. Preferableembodiments are defined in the dependent claims.

Specifically, embodiments presented herein aim at achieving an optimizedvisual experience for the user, and to enable the best possibleprerequisites for the performance of eye tracking or gaze tracking.

These aims are achieved by providing solutions for identifying if ahead-mounted device is positioned correctly at the head of a user.According to some embodiments, the solutions provided herein go evenfarther by, if head-mounted device is incorrectly positioned, doingsomething about it, by for example letting the user and/or system knowabout the problem; guiding the user on how to reposition thehead-mounted device; and/or provide automatic repositioning.

In a first aspect, there are provided embodiments of a method fordetermining if a head-mounted device for extended reality (XR) iscorrectly positioned on a user, the method comprising performing eyetracking, using processing circuitry, by: estimating, based on a firstimage of a first eye of the user, the position of a pupil of the firsteye. The method further comprises determining, using the processingcircuitry, if the head-mounted device is correctly positioned on theuser, by determining whether the estimated position of the pupil of thefirst eye is within a predetermined allowable area in the first image.If the determined position of the pupil of the first eye is inside thepredetermined allowable area of the first image the method comprisesconcluding that the head-mounted device is correctly positioned on theuser. Alternatively, if the determined position of the pupil of thefirst eye is outside the predetermined allowable area of the first imagethe method comprises concluding that the head-mounted device isincorrectly positioned on the user.

According to a second aspect, there are provided embodiments of a systemfor determining if a head-mounted device for extended reality (XR) isincorrectly positioned on a user, the system comprising processingcircuitry configured to perform eye tracking, by estimating, based on afirst image of a first eye of the user, the position of a pupil of thefirst eye. The processing circuitry is further configured to determineif the head-mounted device is correctly positioned on the user, bydetermining whether the estimated position of the pupil of the first eyeis within a predetermined allowable area in the first image. If thedetermined position of the pupil of the first eye is inside thepredetermined allowable area of the first image, the processingcircuitry is configured to conclude that the head-mounted device iscorrectly positioned on the user. If the determined position of thepupil of the first eye is outside the predetermined allowable area ofthe first image the processing circuitry is configured to conclude thatthe head-mounted device is incorrectly positioned on the user.

In a third aspect, there is provided a non-transitory computer-readablestorage medium storing instructions which, when executed by processingcircuitry of a system, cause the system to: perform eye tracking, by:estimating, based on an image of a first eye of the user, the positionof the pupil of the first eye; and determine if the head-mounted deviceis correctly positioned on the user, by: determining whether theestimated position of the pupil of the first eye is within apredetermined allowable area in the first image; and if the determinedposition of the pupil of the first eye is inside the predeterminedallowable area of the first image: concluding that the head-mounteddevice is correctly positioned on the user, or if the determinedposition of the pupil of the first eye is outside the predeterminedallowable area of the first image: concluding that the head-mounteddevice is incorrectly positioned on the user.

The effects and/or advantages presented in the present disclosure forembodiments of the method according to the first aspect may also applyto corresponding embodiments of the non-transitory computer-readablestorage medium according to the third aspect.

It is noted that embodiments of the present disclosure relate to allpossible combinations of features recited in the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

In the following, example embodiments will be described in greaterdetail with reference to the accompanying drawings, in which:

FIG. 1 a is a front view of an eye;

FIG. 1 b shows an image of the eye of FIG. 1 a;

FIG. 1 c shows an image of the eye of FIG. 1 a;

FIG. 2 is a schematic overview of a system, according to one or moreembodiments;

FIG. 3 a is a flow chart of a method for determining if a head-mounteddevice for extended reality (XR) is correctly positioned on a user,according to one or more embodiments;

FIGS. 3 b to 3 e are flow charts showing embodiments of the method ofFIG. 3 a;

FIG. 4 is a cross sectional view of the eye from FIG. 1 a from the sideof the eye;

FIG. 5 shows a first image of the eye of FIG. 1 a and a second image ofa second eye;

FIG. 6 shows a schematic view of a pair of eyes of a user of ahead-mounted device, and the head-mounted device;

FIG. 7 a shows a schematic view of an eye and a head-mounted device,according to one or more embodiments; and

FIG. 7 b shows a schematic view of an eye and an image onto which thegaze of the eye is directed.

All the figures are schematic, not necessarily to scale, and generallyonly show parts which are necessary in order to elucidate the respectiveembodiments, whereas other parts may be omitted or merely suggested. Anyreference number appearing in multiple drawings refers to the sameobject or feature throughout the drawings, unless otherwise indicated.

DETAILED DESCRIPTION Introduction

Embodiments of the present disclosure aim at determining if a user iswearing a head-mounted device correctly. Some embodiments herein furtherinclude, upon determination that the head-mounted device is incorrectlypositioned on the user, performing a position correction procedure. Theposition correction procedure may comprise informing the user, or thesystem, or both, about the determined incorrect positioning of thehead-mounted device, and/or in applicable cases performing automaticposition correction.

By a correct position of head-mounted device on a user, we mean that thehead-mounted device is positioned on the head of the user such that theoptics of the head-mounted device that are arranged to enable the userto experience VR, AR, MR or other XR content, as well as see any imagesof the surroundings and/or see the real world surroundings, is optimallypositioned in relation to the user's field of view. Of course, optimallypositioned is to be understood as being in the optimal position, or in aposition that is not the optimal position, but within an allowabletolerance of the optimal position. This leads to the user's gaze beingdirected at the center, or allowably close to the center, of an imagepresented by the head-mounted device.

To obtain a correct position of the head-mounted device on a user, twodifferent aspects are considered.

-   -   1. Is there to big a difference in distance between the user's        eyes and the lens cups of the head-mounted device? In other        words, does the lens cup distance need to be adjusted? and    -   2. is the field of view (FOV) of the user aligned with optics of        the head-mounted device? In other words, are the        If the answer to one or both of these questions, for one or both        eyes of the user, is yes, then it can be concluded that the        head-mounted device is incorrectly positioned on the user.

Embodiments described herein, in connection with the figures, presentsolutions on how to find reliable answers to one or both of thequestions above. Specifically, embodiments of the method (andcorresponding system and non-transitory computer readable medium)presented in connection with FIGS. 3 d and 3 e are directed at finding areliable answer to question 1, while embodiments of the method (andcorresponding system and non-transitory computer readable medium)presented in connection with FIGS. 3 b and 3 c are directed at finding areliable answer to question 2. As explained herein, a combination ofmethod embodiments is conceivable. The combination may mean apossibility to better answer one of the questions, or to answer bothquestions, thereby achieving an improved answer to the question ofwhether the head-mounted device is correctly positioned on the user ornot.

By ensuring that the head-mounted device is correctly positioned on theuser, we ensure that any image content displayed to the user will be ofthe highest possible quality. At the same time, any camera sensors ofeye tracking functionality in the head-mounted device, or in a systemaccording to embodiments presented herein, will also be able to obtainoptimized images of the eyes of the user, since the user's gaze when theuser looks “straight ahead” will, for each eye, be directed at orsufficiently close to the center of the image depicting the eye. Inother words, the FOV of the user will be aligned with the optimalposition, “sweet spot”, of the optics of the head-mounted device.

Embodiments herein relate to a head-mounted device for virtual reality(VR), augmented reality (AR), mixed reality (MR) or other extendedreality (XR) applications. In other words, in the context of the presentdisclosure, at least the techniques referred to as VR, AR and MR areincluded in the concept XR.

Throughout the present disclosure, the term head-mounted device refersto a device adapted to be worn at the head of a user. The head-mounteddevice may comprise display optics. The display optics is to beunderstood as comprising any optics suitable for generating and/ordisplaying 2D image data, 3D image data, graphical data, holographicdata or other content that may be presented to a user/wearer of thehead-mounted device to convey a VR, AR, MR or other XR experience. Inaddition to the actual display optics, the head-mounted device typicallyalso includes other components. Such other components may for exampleinclude circuits for powering the head-mounted device, sensors fordetecting motion of the head-mounted device, eye or gaze trackingequipment, or a casing for protecting components of the head-mounteddevice. In other words, the term head-mounted device may, but should notnecessarily, be construed as only referring to the actual display opticsintended to be arranged in front of an eye of the user, or in front ofboth eyes of the user.

The term eye tracking as used herein may be understood as comprising:tracking or observing actual parts of an eye, in the real world, in a 3Dmodel of the eye, in a 2D image depicting the eye; or determining whatthe eye is tracking or gazing towards. Determination of what the eye istracking or gazing towards may also be referred to as gaze tracking.

Any embodiment described herein as referring to one eye, for instance afirst eye, of a user is of course equally applicable to the any of theuser's eyes, and may also be performed for both the eyes of a user inparallel, or consecutively.

Throughout the present disclosure, the term obtaining information may beunderstood as receiving information, in a push fashion, and/orretrieving information, in a pull fashion.

Head-mounted device position determination methods, head-mounted deviceposition determination systems, and associated storage media will bedescribed below with reference to FIG. 1 b -8. First, certain featuresof an eye will be described with reference to FIGS. 1 a and 4.

FIG. 1 a is a front view of an eye 100. FIG. 4 is a cross sectional viewof the eye 100 from the side of the eye 100. While FIG. 4 shows more orless the entire eye 100, the front view presented in FIG. 1 a only showsthose parts of the eye 100 which are typically visible from in front ofa person's face. The eye 100 has a cornea 101 and a pupil 102, with apupil center 103 and a pupil edge 104.

System Architecture

FIG. 2 is a schematic overview of a system 200, according to one or moreembodiments, for determining if a head-mounted device 260 for extendedreality (XR) is incorrectly positioned on a user 270. The system 200comprises processing circuitry 210 configured to perform eye tracking toestimate a position of a part of a first eye 100 of the user 270, andconfigured to determine if the head-mounted device 260 is correctlypositioned on the user 270, based on the estimated position of the partof the first eye 100. The system 200 may further comprise, or becommunicatively connected to, a display 720.

The processing circuitry 210 may in some non-limiting embodimentscomprise, or be communicatively connected to, eye tracking equipment220, for example in the form of one or more illuminators forilluminating the eye 100 of the user 270 and one or more cameras forcapturing images of the eye 100 while the eye 100 looks/gazes at thedisplay 720, or any other suitable eye tracking equipment known in theart. The may be communicatively connected to the eye tracking equipment,for example via a wired or wireless connection. The illuminators may forexample be infrared or near infrared illuminators, for example in theform of light emitting diodes (LEDs). However, other types ofilluminators may also be envisaged. The cameras may for example becharged-coupled device (CCD) cameras or Complementary Metal OxideSemiconductor (CMOS) cameras. However, other types of cameras may alsobe envisaged.

The display 720 may for example be a liquid-crystal display (LCD) or aLED display. However, other types of displays may also be envisaged. Thedisplay may for example be flat or curved. The display 720 may forexample be placed in front of one of the user's eyes. Separate displays720 may be employed for the left and right eyes. Separate eye trackingequipment 220 (such as illuminators and cameras) may for example beemployed for the left and right eyes.

The processing circuitry 210 may be employed for eye tracking for botheyes, or there may be separate processing circuitry 210 for the left andright eyes. The system 200 may for example perform eye tracking toestimate a position of a part of an eye for the left and right eyesseparately, and may then determine if the head-mounted device 260 iscorrectly positioned on the user 270, based on the estimated position ofthe part of both eyes. In a non-limiting example, the processingcircuitry 210 may be configured to determine that the head-mounteddevice 260 is incorrectly positioned on the user 270 if it has beendetermined that the head-mounted device 260 is incorrectly positioned onthe user 270 for at least one of the user's eyes.

The processing circuitry 210 may for example comprise one or moreprocessors. The processor(s) may for example be application-specificintegrated circuits (ASIC) configured to perform a specific eye trackingand position determination method. Alternatively, the processor(s) maybe configured to execute instructions (for example in the form of acomputer program) stored in one or more memories 240. Such a memory 240may for example be comprised in the system 200, or may be external to(for example located remotely from) the system 200. The memory 240 maystore instructions for causing the system 200 to perform a methodaccording to any of the embodiments presented in connection with FIGS. 3a to 3 e.

The processing circuitry 210 may in one or more embodiment be configuredto perform any or all of the method embodiments described in connectionwith FIGS. 3 a to 3 e.

It will be appreciated that the system 200 described above withreference to FIG. 2 is provided as an example, and that many othersystems may be envisaged. For example, the system 200 may consist onlyof the processing circuitry 210. The display 720 may for example becomprised in the system 200, or may be regarded as separate from thesystem 200.

Eye tracking performed by systems such as the system 200 in FIG. 2typically employ an eye model. This eye model is calibrated toproperties of the individual user's eyes.

A user 270 wearing a head-mounted device 260 is schematicallyillustrated next to the system 200 in FIG. 2 . The system 200 accordingto any embodiment presented herein may be incorporated in, orcommunicatively connected to, such a head-mounted device 260.

Method Embodiments

In the following, method embodiments will be described in connectionwith FIGS. 3 a to 7 b.

FIG. 3 a shows embodiments of a method for determining if thehead-mounted device 260 for extended reality (XR) is correctlypositioned on the user 270, the method comprising:

In step 310: performing eye tracking, using processing circuitry 210, toestimate a position of a part of a first eye 100 of the user 270.

In step 320: determining, using the processing circuitry 210, if thehead-mounted device 260 is correctly positioned on the user 270, basedon the estimated position of the part of the first eye 100.

In an optional step 330: if the head-mounted device 260 is determined tobe incorrectly positioned on the user 270, performing a positioncorrection procedure.

In the context of the present disclosure, performing the positioncorrection procedure may comprise any or all of the following:

-   -   if the lens cup distance d_(LENS_CUP) has been determined to be        too small or too large, performing motorized position correction        of the lens cup distance d_(LENS_CUP);    -   notifying the system 200 or the user 270 about the determined        incorrect position of the head-mounted device 260; or    -   activating a positioning guide, which positioning guide provides        guidance on steps to be taken for manual correction of the        position of the head-mounted device 260.

Activating a positioning guide may include displaying, presenting orotherwise communicating repositioning information or steps to be takento the user 270. The repositioning information or steps to be taken areconfigured to guide and thereby enabling the user 270 to improve theposition of the head-mounted device 260.

Different embodiments of the method shown in FIG. 3 a will now bepresented in connection with FIG. 3 b to FIG. 3 e.

In some embodiments, step 310 of performing eye tracking, usingprocessing circuitry 210, and/or 320 of determining if the head-mounteddevice 260 is correctly positioned on the user 270, using processingcircuitry 210, may comprise the sub-steps shown in FIG. 3 b , inconnection with FIG. 1 b and FIG. 1 c . In these embodiments, the methodcomprises:

In step 311: estimating, based on a first image 110 of the first eye 100of the user 270, the position of the pupil 102 of the first eye 100 intwo dimensions.

The position of the pupil 102 may be defined as a two dimensional (2D),or three dimensional (3D), coordinate described according to anysuitable coordinate system. For the purposes of step 310, the firstimage 110 is typically a 2D image, whereby the position of the pupil 102is defined in 2D. For example, the position of the pupil 102 may bedefined as a 2D coordinate according to a coordinate system defined inrelation to the first image 110 and/or the camera used for capturing thefirst image 110, for example the coordinate system 530 illustrated inFIG. 5 . Alternatively, the position of the pupil 102 may be defined asa group of two or more 2D coordinates comprised within the area of thepupil as depicted in the first image 110.

In some embodiments, the method may comprise obtaining the image 110, byreceiving or retrieving the first image 110 from a memory or camera (notshown in the figures) incorporated in or external to the system 200.

In some embodiments, the position of the pupil 102 may in the context ofstep 310 be approximated as the position of the pupil center 103. Insome embodiments, the position of the pupil 102 in the context of step310 may be derived from one or more coordinates located on the pupiledge 104. In some embodiments, the position of the pupil 102 may in thecontext of step 310 be approximated as the position of the center of theiris (not shown in the figures).

In step 321: determining whether the estimated position of the pupil 102of the first eye 100 is within a predetermined allowable area 120 in thefirst image 110.

The predetermined allowable area 120 is typically smaller in at leastthe x direction or y direction compared to the first image 110, asillustrated in FIGS. 1 b and 1 c.

If the determined position of the pupil 102 of the first eye 100 isinside the predetermined allowable area 120 of the first image 110, themethod is continued in step 322.

If the determined position of the pupil 102 of the first eye 100 isoutside the predetermined allowable area 120 of the first image 110, themethod is continued in step 323.

If the determined position of the pupil 102 of the first eye 100 isdefined as a single 2D coordinate, the position of the pupil 102 of thefirst eye 100 being outside the predetermined allowable area 120 of thefirst image 110 is to be understood as the coordinate being locatedoutside area 120.

If the determined position of the pupil 102 of the first eye 100 isdefined as a group of two or more 2D coordinates comprised within thearea of the pupil 102 as depicted in the first image 110, the positionof the pupil 102 of the first eye 100 being outside the predeterminedallowable area 120 of the first image 110 is to be understood as a part,or all, of the group of coordinates being located outside area 120.

In step 322: concluding that the head-mounted device 260 is correctlypositioned on the user 270.

In step 323: concluding that the head-mounted device 260 is incorrectlypositioned on the user 270.

Optionally, the method of FIG. 3 , according to one or more embodiments,is performed repeatedly, starting again from step 310 when step 320 iscompleted. This option is illustrated by the dashed arrow in FIG. 3 .The method of FIG. 3 may also, according to one or more otherembodiment, be performed only once, at certain preset time intervals, oraccording to other rules specified in the system setup.

In some embodiments, step 320 of determining, using the processingcircuitry 210, if the head-mounted device 260 is correctly positioned onthe user 270 may further comprise the sub-steps of obtaining a gazeangle β for the first eye 100 and comparing the obtained gaze angle β toa preset gaze angle threshold value T_(GA). If the obtained gaze angle βis below the preset gaze angle threshold value T_(GA), the method thencontinues with the steps 310 and 320 of performing eye tracking anddetermining if the head-mounted device 260 is correctly positioned onthe user 270. If the obtained gaze angle β is not below the preset gazeangle threshold value T_(GA), the method may continue by repeating thesub-steps of obtaining the gaze angle β for the first eye 100 andcomparing the obtained gaze angle β to the preset gaze angle thresholdvalue T_(GA). Obtaining the gaze angle β may be performed separately forboth the first and the second eye 100, 600 of the user 270.

Now referring to FIG. 7 a and FIG. 7 b , the gaze angle β may be definedas the angle between an optical axis 710 of the head mounted device 260and a gaze direction 130 of the first eye 100 of the user 270, whereinthe optical axis 710 of the head mounted device 260 is defined as avector passing through a focal point 740 of the first eye 100 and anorigin of coordinates in an internal coordinate system 630 of the headmounted device 260, and wherein the gaze direction 130 of the first eye100 of the user 270 is defined as a vector passing through the focalpoint 740 of the first eye 100 and a gaze point 750 of the first eye 100at the display 720. The focal point 740 may also be referred to as gazeorigin and typically refers to the center of the eye 100, the center ofthe eye ball of the eye 100, or the center of the cornea 101 of the eye100. The internal coordinate system may for example be the coordinatesystem 630, illustrated from different views in FIGS. 6, 7 a and 7 b.

In other words, the method may comprise, once or iteratively, obtainingthe gaze angle β of an eye of the user and comparing it to a presetthreshold. If the gaze angle β is below the preset gaze angle thresholdvalue T_(GA), this means that the gaze of the user 270, or specificallythe gaze of the first eye 100 of the user 270, is directed at the center730 of an image displayed on the display 720 of the head-mounted device260, or at a point in the image that is within a preset allowabletolerance from the center 730. If the gaze angle β is instead not belowthe preset gaze angle threshold value T_(GA), the gaze of the user 270,or specifically the gaze of the first eye 100 of the user 270, isdirected at a point too far from the center 730 of the image. If this isthe case, for instance if the user 270 is looking at an upper rightcorner of the display 720 instead of gazing straight ahead, which mayalso referred to as gazing or looking in the forward direction, there isno need to perform steps 310 and 320, since the result will not bereliable if the user 270 is not gazing straight ahead in the directionof an optical axis 710 of the head mounted device 260, or at least closeto this direction. The optical axis 710 of the head mounted device 260may as illustrated in FIG. 7 a extend along the z-axis of the internalcoordinate system 630 of the head-mounted device 260, whereby theoptical axis 710 always extends in the forward direction as seen fromthe user 270 gazing towards the display 720 of the head-mounted device260. Therefore, checking the gaze angle β and comparing it to the presetgaze angle threshold value T_(GA) may further improve the performanceand reliability of one or more embodiments presented herein. In someembodiments, there may be defined an optical axis 710, and hence alsothe internal coordinate system 630, for each of the eyes of the user270, wherein each optical axis 710 extends from one eye towards thedisplay 720 arranged to be viewed by said eye. For example, first andsecond head-mounted displays may be arranged in, or in connection two, afirst and second lens cups 610, 620, respectively, comprised in thehead-mounted device 260. An example of such lens cups, 610, 620,comprised in a head-mounted device 260 is schematically illustrated inFIG. 6 .

The method according to any embodiment described in connection withsteps 310 and 320, optionally including one or more sub-steps, may beperformed separately for both the first and second eyes 100, 600 of theuser 270, and it may be concluded that the head-mounted device 260 isincorrectly positioned on the user 270 if this is true for at least oneof the first and second eye.

In some embodiments, step 310 of performing eye tracking, usingprocessing circuitry 210, may comprise the sub-steps shown in FIG. 3 c .In these embodiments, the method comprises:

In step 312: obtaining an eye rotation center 105 for the first eye 100,wherein the eye rotation center 105 defines a point where all gazedirections of the first eye 100 converge behind the cornea 101 of thefirst eye 100.

Obtaining the eye rotation center 105 for the first eye 100 may in someembodiments mean retrieving or receiving a previously determined eyerotation center from the memory 240.

In other embodiments, obtaining the eye rotation center 105 for thefirst eye 100 may be understood as determining the eye rotation center105, by the processing circuitry 210 of the system 200, by: estimating,for each image in a set of multiple images of the first eye 100 of theuser 270, the position of the cornea 101, or the center of the cornea101, and a gaze direction 130 for the first eye 100; and estimating theeye rotation center 105 as the point where all gaze directions 130converge behind the cornea 101 of the first eye 100.

In one or more embodiments, estimating the position of the cornea 101,or the center of the cornea 101, determining the gaze direction 130 andestimating the eye rotation center 105 may be performed separately forboth the first and the second eye 100, 600 of the user 270.

Determination of the eye rotation center 105 may be performed using anysuitable eye tracking method known in the art, e.g. pupil-center-corneareflection (PCCR), corner of eye to pupil, machine learning models, etc.

In step 313: for a given gaze direction 130, estimating the position ofthe pupil 102 of the first eye 100 in three dimensions, based on theobtained eye rotation center 105.

In step 314: estimating the position of the pupil 102 of the first eye100 in two dimensions, based on the estimated position of the pupil 102of the first eye 100 in three dimensions.

In one or more embodiments of the method of FIG. 3 c , estimating theposition of the pupil 102 of the first eye 100, in two dimensions ofstep 311, is based only on the estimated position of the pupil 102 ofthe first eye 100 in three dimensions. Alternatively, estimating theposition of the pupil 102 of the first eye 100, in two dimensions ofstep 311, is further based on the estimated position of the pupil 102 ofthe first eye 100 in three dimensions. In other words, the methodembodiments shown in FIG. 3 b and FIG. 3 c may be combined. In someembodiments, the method may comprise determining, based on an estimatedradius of the eye rotation center 105, where the pupil 102 of the firsteye 100 would be positioned if the gaze direction 130 is straightforward, or in other words aligned with the optical axis 710 of the headmounted device 260. Thus, we can, for any known gaze angle β, includingwhat in this context is considered to be very large gaze angles β,adjust for pupil movement in the 2D image due to the gaze angle β. Inother words, the method may comprise adjusting for pupil movement in a2D image depicting the first eye 100 based on the eye rotation center105 and an obtained gaze angle β. Further, since we can always estimatewhere the pupil 102 would be positioned if the gaze direction 130 isstraight forward, or aligned with the optical axis 710 of the headmounted device 260, we can reduce the predetermined allowable area 120for gaze direction estimates straight forward. In other words, themethod may comprise reducing the predetermined allowable area 120 if thegaze direction 130 is aligned with, or within a preset allowabletolerance of, the optical axis 710 of the head mounted device 260.

The embodiments of method step 310 presented in connection with FIG. 3 cmay be combined with any or all of the embodiments of method step 320presented herein.

In some embodiments, step 310 of performing eye tracking, usingprocessing circuitry 210, and/or step 320 of determining if thehead-mounted device 260 is correctly positioned on the user 270, usingprocessing circuitry 210, may comprise the sub-steps shown in FIG. 3 d .In these embodiments, the method comprises:

In step 315: perform eye tracking by estimating, based on a second image510 of a second eye of the user 270, a position of a pupil 602 of thesecond eye in 2D.

An example of such a second image 510 is shown in FIG. 5 , which shows,in a front view of at least a pupil 602 of the second eye, with a pupilcenter 603 and a pupil edge 604.

In one or more embodiments, each of the positions of the pupils 102, 602of the first and second eye, respectively, are defined according to acommon coordinate system. A non-limiting example of such a commoncoordinate system 530, in 2D, is illustrated in FIG. 5 , defined inrelation to the first and/or second image 110, 510 and/or the cameraused for capturing the first and/or second image 110, 510. The positionof the pupil 602 may be defined as a 2D coordinate according to thecommon coordinate system. Alternatively, the position of the pupil 602may be defined as a group of two or more 2D coordinates comprised withinthe area of the pupil 602 as depicted in the second image 510.

In some embodiments, step 315 may comprise obtaining the second image510, by receiving or retrieving the first image 110 from the memory 240or camera (not shown in the figures) incorporated in or external to thesystem 200.

In some embodiments, the position of the pupils 102, 602 of the firstand second eyes, respectively, may be approximated as the position ofthe center 103 of the first pupil 102, and the center 603 of the secondpupil 602, respectively. In other embodiments, the position of thepupils 102, 602 of the first and second eyes, respectively, may bederived from one or more coordinates located on the pupil edge 104 ofthe first pupil 102 and the pupil edge 604 of the second pupil 602,respectively. In some embodiments, the position of the pupils 102, 602of the first and second eyes 100, 600, respectively, may be approximatedas the position of the center of the iris of the first pupil 102, andthe center of the iris of the second pupil 602, respectively.

In step 316: determining an eye separation distance d_(EYES) as thedifference between the position of the pupil 102 of the first eye 100and the position of the pupil 602 of the second eye.

In step 324: determining whether the absolute value of the determinedeye separation distance d_(EYES) is above a preset eye separationthreshold value T_(EYES).

If the absolute value of the determined eye separation distance d_(EYES)is above the preset eye separation threshold value T_(EYES), the methodis continued in step 325.

If the absolute value of the determined eye separation distance d_(EYES)is not above the preset eye separation threshold value T_(EYES), themethod is continued in step 326.

In some embodiments, the eye separation distance d_(EYES) may bedetermined in one dimension, for example as the distance in the xdirection of the example coordinate system 530. In other embodiments,the eye separation distance d_(EYES) may be determined in twodimensions, for example as the distance in the x direction and the ydirection of the example coordinate system 530.

In step 325: concluding that the head-mounted device 260 is incorrectlypositioned on the user 270.

In other words, step 325 concluding that there is a difference betweenthe distance between the eyes of the user 270 and the lens cups 610, 620that exceeds the preset threshold value T_(EYES), wherein T_(EYES)defines the limit for when the difference is too big for the system 200to provide an acceptable visual experience for the user 270. In yetother words, if T_(EYES) is exceeded, the lens cup distance d_(LENS_CUP)needs to be adjusted, because the lens cups 610, 620 are not alignedwith the user's eyes and FOV.

In step 326: concluding that the head-mounted device 260 is correctlypositioned on the user 270.

The embodiments steps 315, 316 and 324 to 326 described in connectionwith FIG. 3 d may be combined with any or all of the other embodimentsof method steps 310 and/or 320 presented herein. Thereby, alternativeand possibly further improved manners of determining if the head-mounteddevice 260 is correctly positioned on a user 270 may be obtained.

In some embodiments, step 310 of performing eye tracking, usingprocessing circuitry 210, and/or step 320 of determining if thehead-mounted device 260 is correctly positioned on the user 270, usingprocessing circuitry 210, may comprise sub-steps shown in FIG. 3 e . Inthese embodiments, the method comprises:

In step 317: estimating a lens cup separation distance d_(LENS_CUP) asthe difference between the position of the center 611 of the first lenscup 610 of the head-mounted device 260 and the position of the center621 of the second lens cup 620 of the head-mounted device 260, in 3D.

The position of the center 611 of the first lens cup 610 of thehead-mounted device 260 and the position of the center 621 of the secondlens cup 620 of the head-mounted device 260 may be obtained in in anysuitable manner, for example by being received or retrieved from thememory 240 or other processing circuitry 210, or calculated during useof the head-mounted device 260.

In step 318: performing eye tracking by estimating an eye separationdistance d_(EYES) as the difference between the position of the pupil102 of the first eye 100 of the user 270 and the position of the pupil602 of the second eye 600 of the user 270, in 3D.

Each of the positions of the centers 611, 621 of the first and secondlens cup 610, 620, respectively, and each of the pupils 102, 602 of thefirst and second eye 100, 600 of the user 270, respectively, may bedefined according to a common coordinate system, for example thecoordinate system 630.

Alternatively, instead of using the pupil position, step 318 maycomprise using for instance pupil centers 103, 603, cornea centers, iriscenters or eye ball centers of the first and second eyes, respectively,to estimate the eye separation.

It will be appreciated that steps 317 and 318 may be performed in anyorder, subsequently or in parallel.

In step 319: calculating the difference d_(DIFF) between the lens cupseparation distance d_(LENS_CUP) and the eye separation distanced_(EYES).

In one or more embodiment, the difference d_(DIFF) is defined andcalculated in 3D.

In step 327: determining whether the calculated difference d_(DIFF)between the lens cup separation distance d_(LENS_CUP) and the eyeseparation distance d_(EYES) is above a preset difference thresholdvalue T_(DIFF).

If the absolute value of the determined eye separation distance d_(EYES)is above the preset eye separation threshold value T_(EYES), the methodis continued in step 328.

If the absolute value of the determined eye separation distance d_(EYES)is not above the preset eye separation threshold value T_(EYES), themethod is continued in step 329.

In step 328: concluding that the head-mounted device 260 is incorrectlypositioned on the user 270.

In other words, step 328 comprises concluding that there is a differencebetween the distance between the eyes of the user 270 and the lens cups610, 620 that exceeds the preset threshold value T_(EYES), whereinT_(EYES) defines the limit for when the difference is too big for thesystem 200 to provide an acceptable visual experience for the user 270.In yet other words, if T_(EYES) is exceeded, the lens cup distanced_(LENS_CUP) needs to be adjusted, because the lens cups 610, 620 arenot aligned with the user's eyes and FOV.

In step 329: concluding that the head-mounted device 260 is correctlypositioned on the user 270.

The embodiments steps 315, 316 and 324 to 326 described in connectionwith FIG. 3 d may be combined with any or all of the other embodimentsof method steps 310 and/or 320 presented herein. Thereby, alternativeand possibly further improved manners of determining if the head-mounteddevice 260 is correctly positioned on a user 270 may be obtained.

Further Embodiments

In one or more embodiment, there is provided a non-transitorycomputer-readable storage medium storing instructions which, whenexecuted by processing circuitry 210 of the system 200, cause the system200 to perform the method of any of the embodiments presented herein.The non-transitory computer-readable storage medium may storeinstructions which, when executed by processing circuitry 210 of thesystem 200, cause the system 200 to perform eye tracking, by:estimating, based on an image of the first eye 100 of the user, theposition of the pupil 102 of the first eye 100; and determine if thehead-mounted device 260 is correctly positioned on the user 270, by:determining whether the estimated position of the pupil 102 of the firsteye 100 is within a predetermined allowable area 120 in the first image110; and if the determined position of the pupil 102 of the first eye100 is inside the predetermined allowable area 120 of the first image110: concluding that the head-mounted device 260 is correctly positionedon the user, or if the determined position of the pupil 102 of the firsteye 100 is outside the predetermined allowable area 120 of the firstimage 110: concluding that the head-mounted device 260 is incorrectlypositioned on the user 270. In some embodiments, the non-transitorycomputer-readable storage medium may further storing instructions which,when executed by processing circuitry 210 of the system 200, cause thesystem 200 to perform a position correction procedure if thehead-mounted device 260 is determined to be incorrectly positioned onthe user 270.

The non-transitory computer-readable storage medium may in someembodiments store instructions which, when executed by processingcircuitry 210 of the system 200, cause the system 200 to perform themethod as defined in any of the method disclosed herein (in other words,in the claims, the summary, or the detailed description).

The non-transitory computer-readable storage medium may for example beprovided in a computer program product. In other words, a computerprogram product may for example comprise a non-transitorycomputer-readable storage medium storing instructions which, whenexecuted by the processing circuitry 210 of the system 200, cause thesystem 200 to perform the method as defined in any of the methodembodiments

As described above with reference to FIG. 2 , the storage medium neednot necessarily be comprised in the system 200.

The person skilled in the art realizes that the present invention is byno means limited to the preferred embodiments described above. On thecontrary, many modifications and variations are possible within thescope of the appended claims. For example, the embodiments describedabove with reference to FIGS. 3 a to 7 b may, as explained herein, becombined to form further embodiments. Further, it will be appreciatedthat the system 200 shown in FIG. 2 is merely intended as an example,and that other systems may also perform the methods described above withreference to FIGS. 3 a to 3 e.

It will be appreciated that the processing circuitry 210 (or aprocessor) may comprise a combination of one or more of amicroprocessor, controller, microcontroller, central processing unit,digital signal processor, application-specific integrated circuit, fieldprogrammable gate array, or any other suitable computing device,resource, or combination of hardware, software and/or encoded logicoperable to provide computer functionality, either alone or inconjunction with other computer components (such as a memory or storagemedium).

It will also be appreciated that a memory or storage medium (or acomputer-readable medium) may comprise any form of volatile ornon-volatile computer readable memory including, without limitation,persistent storage, solid-state memory, remotely mounted memory,magnetic media, optical media, random access memory (RAM), read-onlymemory (ROM), mass storage media (for example, a hard disk), removablestorage media (for example, a flash drive, a Compact Disk (CD) or aDigital Video Disk (DVD)), and/or any other volatile or non-volatile,non-transitory device readable and/or computer-executable memory devicesthat store information, data, and/or instructions that may be used by aprocessor or processing circuitry.

Additionally, variations to the disclosed embodiments can be understoodand effected by those skilled in the art in practicing the claimedinvention, from a study of the drawings, the disclosure, and theappended claims. In the claims, the word “comprising” does not excludeother elements or steps, and the indefinite article “a” or “an” does notexclude a plurality. In the claims, the word “or” is not to beinterpreted as an exclusive or (sometimes referred to as “XOR”). On thecontrary, expressions such as “A or B” covers all the cases “A and notB”, “B and not A” and “A and B”, unless otherwise indicated. The merefact that certain measures are recited in mutually different dependentclaims does not indicate that a combination of these measures cannot beused to advantage. Any reference signs in the claims should not beconstrued as limiting the scope.

The invention claimed is:
 1. A method for determining if a head-mounteddevice for extended reality (XR) is correctly positioned on a user, themethod comprising: performing eye tracking, using processing circuitry,by estimating, based on a first image of a first eye of the user, theposition of the pupil of the first eye in two dimensions; determining,using the processing circuitry, if the head-mounted device is correctlypositioned on the user, by: determining whether the estimated positionof the pupil of the first eye is within a predetermined allowable areain the first image; and (i) if the determined position of the pupil ofthe first eye is inside the predetermined allowable area of the firstimage: concluding that the head-mounted device is correctly positionedon the user, or (ii) if the determined position of the pupil of thefirst eye is outside the predetermined allowable area of the firstimage: concluding that the head-mounted device is incorrectly positionedon the user further comprising using the processing circuitry to:perform eye tracking by estimating, based on a second image of a secondeye of the user, the position of the pupil of the second eye in twodimensions, wherein each of the positions of the pupils of the first andsecond eyes, respectively, are defined according to a common coordinatesystem; determining an eye separation distance (d_(EYES)) as thedifference between the position of the pupil of the first eye and theposition of the pupil of the second eye, wherein determining, using theprocessing circuitry, if the head-mounted device is correctly positionedon the user, further comprises: if the absolute value of the determinedeye separation distance (d_(EYES)) is above a preset eye separationthreshold value (T_(EYES)): concluding that the head-mounted device isincorrectly positioned on the user.
 2. The method of claim 1, whereinthe step of determining, using the processing circuitry, if thehead-mounted device is correctly positioned on the user, furthercomprises: a) obtaining a gaze angle (β) for the first eye; b) comparingthe obtained gaze angle (β) to a preset gaze angle threshold value(T_(GA)); and if the obtained gaze angle (β) is below the preset gazeangle threshold value (T_(GA)), continuing with the steps of performingeye tracking and determining if the head-mounted device is correctlypositioned on the user; or if the obtained gaze angle (β) is not belowthe preset gaze angle threshold value (T_(GA)), returning to step a). 3.The method of claim 2, wherein the obtained gaze angle (β) is defined asthe angle between an optical axis of the head mounted device and a gazedirection of the first eye of the user, wherein the optical axis of thehead mounted device is defined as a vector passing through a focal pointof the first eye and an origin of coordinates in an internal coordinatesystem of the head mounted device, and wherein the gaze direction of thefirst eye of the user is defined as a vector passing through the focalpoint of the first eye and a gaze point of the first eye at the display.4. The method of claim 1, wherein the step of performing eye tracking,using processing circuitry, further comprises: obtaining an eye rotationcenter for the first eye wherein the eye rotation center defines a pointwhere all gaze directions of the first eye converge behind the cornea ofthe first eye; and for a given gaze direction, estimating the positionof the pupil of the first eye in three dimensions, based on the obtainedeye rotation center, wherein estimating the position of the pupil of thefirst eye, in two dimensions is further based on the estimated positionof the pupil of the first eye in three dimensions.
 5. The method ofclaim 1, further comprising: estimating a lens cup separation distance(d_(LENS_CUP)) as the difference between the position of the center of afirst lens cup of the head-mounted device and the position of the centerof a second lens cup of the head-mounted device in three dimensions;performing eye tracking by estimating the eye separation distance(d_(EYES)) as the difference between the position of the pupil of thefirst eye of the user and the position of the pupil of the second eye ofthe user, in three dimensions, wherein each of the positions of thecenters of the first and second lens cups, respectively, and each of thepupils of the first and second eyes of the user, respectively, aredefined according to the common coordinate system; and calculating adifference (d_(DIFF)) between the lens cup separation distance(d_(LENS_CUP)) and the eye separation distance (d_(EYES)); whereindetermining, using the processing circuitry, if the head-mounted deviceis correctly positioned on the user, further comprises: comparing thecalculated difference (d_(DIFF)) between the lens cup separationdistance (d_(LENS_CUP)) and the eye separation distance (d_(EYES)) to apreset difference threshold value (T_(DIFF)); and if the calculateddifference (d_(DIFF)) is above the preset difference threshold value(T_(DIFF)): concluding that the head-mounted device is incorrectlypositioned on the user.
 6. The method of claim 1, further comprising, ifthe head-mounted device is determined to be incorrectly positioned onthe user: performing a position correction procedure.
 7. The method ofclaim 6, wherein performing the position correction procedure comprises,if a lens cup separation distance (d_(LENS_CUP)) has been determined tobe too small or too large, performing motorized position correction ofthe lens cup separation distance (d_(LENS_CUP)).
 8. The method of claim6, wherein performing the position correction procedure comprisesnotifying the system or the user about the determined incorrect positionof the head-mounted device.
 9. The method of claim 6, wherein performingthe position correction procedure comprises initiating a positioningguide, which positioning guide provides guidance on steps to be takenfor manual correction of the position of the head-mounted device.
 10. Asystem for determining if a head-mounted device for extended reality(XR) is incorrectly positioned on a user, the system comprisingprocessing circuitry configured to: perform eye tracking by estimating,based on a first image of a first eye of the user, a position of a pupilof the first eye; and determine if the head-mounted device is correctlypositioned on the user, by: determining whether the estimated positionof the pupil of the first eye is within a predetermined allowable areain the first image; and (i) if the determined position of the pupil ofthe first eye is inside the predetermined allowable area of the firstimage: concluding that the head-mounted device is correctly positionedon the user, or (ii) if the determined position of the pupil of thefirst eye is outside the predetermined allowable area of the firstimage: concluding that the head-mounted device is incorrectly positionedon the user; wherein the processing circuitry is further configured to:perform eye tracking by estimating, based on a second image of a secondeye of the user, the position of the pupil of the second eye in twodimensions, wherein each of the positions of the pupils of the first andsecond eyes, respectively, are defined according to a common coordinatesystem; determine an eye separation distance (d_(EYES)) as thedifference between the position of the pupil of the first eye and theposition of the pupil of the second eye, wherein the processingcircuitry is configured to determine if the head-mounted device iscorrectly positioned on the user by: if the absolute value of thedetermined eye separation distance (d_(EYES)) is above a preset eyeseparation threshold value (T_(EYES)); concluding that the head-mounteddevice is incorrectly positioned on the user.
 11. The system of claim10, wherein the processing circuitry is further configured to: a) obtaina gaze angle (β) for the first eye; b) compare the obtained gaze angle(β) to a preset gaze angle threshold value (T_(GA)); and if the obtainedgaze angle (β) is below the preset gaze angle threshold value (T_(GA)),continue with the steps of performing eye tracking and determining ifthe head-mounted device is correctly positioned on the user; or if theobtained gaze angle (β) is not below the preset gaze angle thresholdvalue (T_(GA)), return to step a).
 12. The system of claim 10, whereinthe processing circuitry is further configured to: perform the eyetracking by obtaining an eye rotation center for the first eye, whereinthe eye rotation center defines a point where all gaze directions of thefirst eye converge behind the cornea of the first eye; and for a givengaze direction, estimating the position of the pupil of the first eye inthree dimensions, based on the obtained eye rotation center, whereinestimating the position of the pupil of the first eye in two dimensions,is further based on the estimated position of the pupil of the first eyein three dimensions.
 13. The system of claim 10, wherein the processingcircuitry is further configured to: estimate a lens cup separationdistance (d_(LENS_CUP)) as the difference between the position of thecenter of a first lens cup of the head-mounted device and the positionof the center of a second lens cup of the head-mounted device; performeye tracking by estimating the eye separation distance (d_(EYES)) as thedifference between the position of the pupil of the first eye of theuser and the position of the pupil of the second eye of the user,wherein each of the positions of the centers of the first and secondlens cups, respectively, and each of the pupils of the first and secondeyes of the user, respectively, are defined according to the commoncoordinate system; calculate a difference (d_(DIFF)) between the lenscup separation distance (d_(LENS_CUP)) and the eye separation distance(d_(EYES)); and compare the calculated difference (d_(DIFF)) between thelens cup separation distance (d_(LENS_CUP)) and the eye separationdistance (d_(EYES)) to a preset difference threshold value (T_(DIFF)),wherein the processing circuitry is further configured to determine ifthe head-mounted device is correctly positioned on the user by: if thecalculated difference (d_(DIFF)) is above the preset differencethreshold value (T_(DIFF)): concluding that the head-mounted device isincorrectly positioned on the user.
 14. The system of claim 10, whereinthe processing circuitry is further configured to perform a positioncorrection procedure if the head-mounted device is determined to beincorrectly positioned on the user.
 15. The system of claim 14, whereinthe processing circuitry is configured to, in the position correctionprocedure: perform motorized position correction of a lens cupseparation distance (d_(LENS_CUP)), if the lens cup separation distance(d_(LENS_CUP)) has been determined to be too small or too large; notifythe system or the user about the determined incorrect position of thehead-mounted device; or initiate a positioning guide, which positioningguide provides guidance on steps to be taken for manual correction ofthe position of the head-mounted device.
 16. The system of claim 10further comprising a head-mounted device.
 17. A non-transitorycomputer-readable storage medium storing instructions which, whenexecuted by processing circuitry of a system cause the system to:perform eye tracking, by estimating, based on an image of a first eye ofthe user, the position of the pupil of the first eye; and determine ifthe head-mounted device is correctly positioned on the user, by:determining whether the estimated position of the pupil of the first eyeis within a predetermined allowable area in the first image; and (i) ifthe determined position of the pupil of the first eye is inside thepredetermined allowable area of the first image: concluding that thehead-mounted device is correctly positioned on the user, or (ii) if thedetermined position of the pupil of the first eye is outside thepredetermined allowable area of the first image: concluding that thehead-mounted device is incorrectly positioned on the user wherein thesystem is further caused to: perform eye tracking by estimating, basedon a second image of a second eye of the user, the position of the pupilof the second eye in two dimensions, wherein each of the positions ofthe pupils of the first and second eyes, respectively, are definedaccording to a common coordinate system; determining an eye separationdistance (d_(EYES)) as the difference between the position of the pupilof the first eye and the position of the pupil of the second eye,wherein determining, using the processing circuitry, if the head-mounteddevice is correctly positioned on the user, further comprises: if theabsolute value of the determined eye separation distance (d_(EYES)) isabove a preset eye separation threshold value (T_(EYES)): concludingthat the head-mounted device is incorrectly positioned on the user. 18.The non-transitory computer-readable storage medium of claim 17, furtherstoring instructions which, when executed by processing circuitry of asystem, cause the system to perform a position correction procedure ifthe head-mounted device is determined to be incorrectly positioned onthe user.